1. Field of the Invention
This invention relates to the field of computer image processing and computer aided design. More specifically, the invention relates to editing three dimensional models by editing two dimensional images
2. Background Art
The size and complexity of three dimensional (3D) digital models is increasing in many applications. Advanced computer aided-design systems running on powerful workstations enable the creation of complex, large models. Furthermore, scanning devices can capture models of up to billions of vertices with accompanying surface properties. The editing of such complex 3D objects is a challenging user interface problem
Fine detailed editing of objects is needed for the design of products requiring aesthetic appeal that are to be produced by programmed machines—i.e., products that have to look good and are going to be produced by automated manufacturing processes based on numerical descriptions of products. Examples of such products include containers for home products like laundry detergent, home appliances like vacuum cleaners or air freshener plug-ins, children's toys, and home furnishings.
Many products are designed by geographically distributed teams. There thus is a need for designers to interact with large data sets that are stored at some distant location, and so there is a need for a “lightweight” object editing interface.
Complex models are increasing used in feature film and computer games. Technicians in these areas need fine control over object editing to create character expressions and to provide realistic props and backgrounds.
Engineers and technicians are no longer the only population that want to edit complex models. Detailed 3D scans of patients may be used by medical doctors planning plastic surgery. Detailed scans can also used by museums and archaeological excavations. Object editing may be required by historians or archaeologists to study and evaluate conjectures about digitized artifacts.
One reason that 3D object editing is difficult is that it is now common for models to be comprised of hundreds of megabytes or more. While numerous simplification methods have been developed, many objects overwhelm rendering systems when displayed with full visual quality. A second reason the problem is difficult is that a user is trying to modify an object in 3D with either a two dimensional (2D) or an imprecise 3D interface device. Large scale, general changes in shape are readily indicated, but fine scale editing over a large region is difficult. In 2D, a 2D gesture by the user needs to be interpreted by the interface as a 3D operation—such as pushing, pulling or cutting an object. In 3D, the response the user has using common haptic devices is still limited compared with true physical modeling modalities such as modeling clay and stone sculpture.
Prior art systems for 2D image painting and editing have evolved over several decades. While a variety of commercial systems are available, they share a common set of user tools. Commercial products have extended these 2D paint tools to painting multiple layers of appearance attributes (color, bumps, shininess etc.) on 3D objects. These 3D paint programs offer the option of a “projection paint” mode that allows the user to export 2D images of an object from an arbitrary view to a 2D image editing program to edit the object texture layers with a full array of 2D imaging tools. The projection paint system then projects the user's texture edits back onto the maps associated with the 3D model.
Recognizing the success of 2D paint programs, researchers in 3D geometric editing have adapted successful tools such as cut-and-paste and other image operations to 3D editing toolkits. However, these systems use the 2D toolkits as inspiration, rather than giving users the ability to edit geometry from within an existing 2D editing system. Some systems allow users to use 2D systems to create 2.5D surfaces by interpreting painted intensity values as heights. These systems are indirect however, because gray-scale coded heights do not correspond to a shaded rendering of an object.
With the interest in recent years in image-based modeling and rendering, many efforts have explored how geometry and images can be beneficially combined. Various researchers have considered how limited geometric information can be used to enhance image editing systems. In Oh, Chen, Dorsey and Durand, “Image-based modeling and photo editing,” Proceedings of SIGGRAPH 2001, pp. 433–442, a system was presented that allows a user to add depth information that is then used in an enhanced photo-editing system. In Seitz and Kutulakos, “Plenotic Image Editing,” Proc. of the 6th International Conference on Computer Vision, 1998, pp. 17–24, the authors describe a method that uses a crude intermediate geometric representation to facilitate the simultaneous editing of multiple views of the same object.
Recently there has also been substantial interest in refining methods from computer vision for geometric object creation, such as the method described in Debevec, Taylor, and Malik, “Modeling and Rendering architecture from photographs: a hybrid geometry-and image-based approach,” Proceedings of SIGGRAPH 1996, pp. 11–20. While methods from computer vision have been successful for creating 3D objects, most methods do not lend themselves to editing 3D objects. It is natural to ask the user to edit a single image of the shape as they want to see it.
The method from computer vision which extracts an object from a natural image is referred to as shape from shading. A wide variety of shape from shading algorithms exist but have not gained popularity in capture systems because they are not robust in the presence of effects such as spatially varying albedo, uncertainty in light source direction and surfaces with discontinuities. In Van Overveld, “Painting gradients: Free-form surface design using shading patterns,” in Graphics Interface 96, 1996, pp. 151–158, Van Overveld makes compelling arguments for an image-based geometric editing system, but rejects the use of shape from shading as too time consuming. Instead, Van Overveld proposes a specialized system in which a user paints gradients directly and is required to define an integrable surface at each step in the editing process. This constrains intermediate edits, and disallows the use of general 2D paint programs.